{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we begin, let us execute the below cell to display information about the NVIDIA® CUDA® driver and the GPUs running on the server by running the `nvidia-smi` command. To do this, execute the cell block below by clicking on it with your mouse, and pressing Ctrl+Enter, or pressing the play button in the toolbar above. You should see some output returned below the grey cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "The **goal** of this lab is to:\n",
    "\n",
    "- Learn how to run the same code on both a multicore CPU and a GPU using standard language parallelism\n",
    "- Understand steps required to make a sequential code parallel using ISO Fortran and ISO C++\n",
    "\n",
    "We do not intend to cover:\n",
    "- Detailed optimization techniques and mapping of standard constructs to CUDA C\n",
    "\n",
    "\n",
    "<details>\n",
    "    <summary markdown=\"span\"><b>ISO C++</b></summary>\n",
    "    \n",
    "# Standard Template Library (STL)\n",
    "    \n",
    "If you are not familiar with STL (Standard Template Library), this section will give you a brief introduction that would be required to understand the usage of STL library for our code.\n",
    "\n",
    "The C++ STL (Standard Template Library) is a powerful set of C++ template classes to provide general-purpose classes and functions with templates that implement many popular and commonly used algorithms and data structures like vectors, lists, queues, and stacks.\n",
    "\n",
    "At the core of the C++ Standard Template Library are following three well-structured components \n",
    "\n",
    "- Containers: Containers are used to manage collections of objects of a certain kind. There are several different types of containers like dequeue, list, vector, map etc.\n",
    "\n",
    "- Algorithms: Algorithms act on containers. They provide the means by which you will perform initialization, sorting, searching, and transforming of the contents of containers.\n",
    "\n",
    "- Iterators: Iterators are used to step through the elements of collections of objects. These collections may be containers or subsets of containers.\n",
    "\n",
    "For our code to make *Pair Calculation* we will be making use of ```vector``` container. The example below will introduce you to the container and how to use an iterator to step through elements of a vector. ```vector``` container (a C++ Standard Template) which is similar to an array with an exception that it automatically handles its own storage requirements in case it grows\n",
    "\n",
    "For our code, we will be making use of ```std::for_each``` algorithm and its sample usage is also shown in the code below:\n",
    "\n",
    "```cpp\n",
    "#include <vector>\n",
    "#include <algorithm>\n",
    "#include <iostream>\n",
    " \n",
    "//Using functor\n",
    "struct Sum\n",
    "{\n",
    "    void operator()(int n) { sum += n; }\n",
    "    int sum{0};\n",
    "};\n",
    " \n",
    "int main()\n",
    "{\n",
    "    std::vector<int> nums{3, 4, 2, 8, 15, 267};\n",
    " \n",
    "    auto print = [](const int& n) { std::cout << \" \" << n; };\n",
    " \n",
    "    std::cout << \"before:\";\n",
    "    std::for_each(nums.cbegin(), nums.cend(), print);\n",
    "    std::cout << '\\n';\n",
    " \n",
    "    std::for_each(nums.begin(), nums.end(), [](int &n){ n++; });\n",
    " \n",
    "    // calls Sum::operator() for each number\n",
    "    Sum s = std::for_each(nums.begin(), nums.end(), Sum());\n",
    " \n",
    "    std::cout << \"after: \";\n",
    "    std::for_each(nums.cbegin(), nums.cend(), print);\n",
    "    std::cout << '\\n';\n",
    "    std::cout << \"sum: \" << s.sum << '\\n';\n",
    "}\n",
    "```\n",
    "\n",
    "To learn more about STL you can read and execute sample codes [here](https://www.tutorialspoint.com/cplusplus/cpp_stl_tutorial.htm).\n",
    "\n",
    "\n",
    "# Parallel STL\n",
    "Starting with C++17, parallelism has become an integral part of the standard itself. Parallel STL is an implementation of the C++ standard library algorithms with support for execution policies, commonly called C++17.\n",
    "\n",
    "C++17 Parallel Standard Library (stdpar) introduces parallel and vector concurrency for standard algorithms. It is important to note that stdpar is a library and not a language extension.\n",
    "\n",
    "\n",
    "## `std::par` Execution Policies\n",
    "\n",
    "\n",
    "Execution Policies define the kind of parallelism that will be applied to parallel algorithms. Most standard algorithms included in STL support execution policies. Defined below are the execution policies:\n",
    "\n",
    "- `std::execution::seq` = sequential\n",
    "    - This execution policy type is used as a unique type to disambiguate parallel algorithm overloading and requires that a parallel algorithm’s execution may not be parallelized.\n",
    "- `std::execution::par` = parallel\n",
    "    - This execution policy type is used as a unique type to disambiguate parallel algorithm overloading and indicate that a parallel algorithm’s execution may be parallelized\n",
    "- `std::execution::par_unseq` = parallel + vectorized\n",
    "    - This execution policy type used as a unique type to disambiguate parallel algorithm overloading and indicate that a parallel algorithm’s execution may be parallelized and vectorized\n",
    "\n",
    "Implementation of execution policies is provided by different compilers from specific vendors. For GPU parallel execution policy we will be making use of the NVIDIA compiler. \n",
    "\n",
    "\n",
    "## Historical Perspective\n",
    "\n",
    "Changes to how the call to _stl_ algorithms changed the new version of C++ standard to incorporate execution policies:\n",
    "\n",
    "**C++98:** \n",
    "```cpp\n",
    "std::sort(c.begin(), c.end()); \n",
    "```\n",
    "**C++17:** \n",
    "```cpp\n",
    "std::sort(std::execution::par, c.begin(), c.end());\n",
    "```\n",
    "\n",
    "We will be using the NVIDIA HPC C++ compiler, NVC++. It supports C++17, C++ Standard Parallelism (stdpar) for NVIDIA GPUs, OpenACC for multicore CPUs and NVIDIA GPUs, and OpenMP for multicore CPUs. No language extensions or non-standard libraries are required to enable GPU acceleration. All data movement between host memory and GPU device memory is performed implicitly and automatically under the control of CUDA [Unified Memory](../GPU_Architecture_Terminologies.ipynb), which means that heap memory is automatically shared between a CPU(Host) and GPU(Device). Stack memory and global memory are not shared. Below given example shows the right allocation and usage of the stdpar.\n",
    "\n",
    "```cpp\n",
    "std::vector<int> v = ...;\n",
    "std::sort(std::execution::par, v.begin(), v.end()); // OK, vector allocates on heap\n",
    "\n",
    "std::array<int, 1024> a = ...;\n",
    "std::sort(std::execution::par, a.begin(), a.end()); // Fails, array stored on the stack\n",
    "```\n",
    "\n",
    "For our code we will be making use of ```std::for_each``` algorithm with support for ```std::execution::par``` execution policy\n",
    "\n",
    "**Counting Iterator**: In our code we will also be using a special iterator ```counting_iterator```. This iterator  represents a pointer into a range of sequentially changing values. This iterator is useful for creating a range filled with a sequence without explicitly storing it in memory. Using ```counting_iterator``` saves memory capacity and bandwidth\n",
    "</details>\n",
    "<br/>\n",
    "    \n",
    "<details>\n",
    "    <summary markdown=\"span\"><b>ISO Fortran</b></summary>\n",
    "    \n",
    "# Fortran Standard Parallelism\n",
    "\n",
    "ISO Standard Fortran 2008 introduced the DO CONCURRENT construct to allow you to express loop-level parallelism, one of the various mechanisms for expressing parallelism directly in the Fortran language. \n",
    "\n",
    "Fortran developers have  been able to accelerate their programs using CUDA Fortran, OpenACC or OpenMP. Now with the support of DO CONCURRENT on GPU with NVIDIA HPC SDK, the compiler automatically accelerates loops using DO CONCURRENT, allowing developers to get the benefit of accelerating  on NVIDIA GPUs using ISO Standard Fortran without any extensions, directives, or non-standard libraries. You can now write standard Fortran, remaining fully portable to other compilers and systems, and still benefit from the full power of NVIDIA GPUs\n",
    "\n",
    "For our code to make *Pair Calculation* all that’s required is expressing loops with DO CONCURRENT. The example below will introduce you to the syntax of DO CONCURRENT \n",
    "\n",
    "Sample vector addition code is shown in code below:\n",
    "\n",
    "```fortran\n",
    "  subroutine vec_addition(x,y,n)\n",
    "    real :: x(:), y(:)\n",
    "    integer :: n, i  \n",
    "    do i = 1, n \n",
    "      y(i) = x(i)+y(i)\n",
    "    enddo  \n",
    "  end subroutine vec_addition\n",
    "```\n",
    "\n",
    "In order to make use of ISO Fortran DO CONCURRENT we need to replace the `do` loop with `do concurrent` as shown in code below\n",
    "\n",
    "```fortran\n",
    "  subroutine vec_addition(x,y,n)\n",
    "    real :: x(:), y(:)\n",
    "    integer :: n, i  \n",
    "    do concurrent (i = 1: n) \n",
    "      y(i) = x(i)+y(i)\n",
    "    enddo  \n",
    "  end subroutine vec_addition\n",
    "```\n",
    "\n",
    "By changing the DO loop to DO CONCURRENT, you are telling the compiler that there are no data dependencies between the n loop iterations. This leaves the compiler free to generate instructions that the iterations can be executed in any order and simultaneously. The compiler parallelizes the loop even if there are data dependencies, resulting in race conditions and likely incorrect results. It’s your responsibility to ensure that the loop is safe to be parallelized.\n",
    "\n",
    "## Nested Loop Parallelism\n",
    "\n",
    "Nested loops are a common code pattern encountered in HPC applications. A simple example might look like the following:\n",
    "\n",
    "```fortran\n",
    "  do i=2, n-1\n",
    "    do j=2, m-1\n",
    "      a(i,j) = w0 * b(i,j) \n",
    "    enddo\n",
    "  enddo\n",
    "```\n",
    "\n",
    "It is straightforward to write such patterns with a single DO CONCURRENT statement, as in the following example. It is easier to read, and the compiler has more information available for optimization.\n",
    "\n",
    "```fortran\n",
    "  do concurrent(i=2 : n-1, j=2 : m-1)\n",
    "    a(i,j) = w0 * b(i,j) \n",
    "  enddo\n",
    "```\n",
    "</details>\n",
    "<br/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, lets start <mark>modifying the original code and add the necessary changes to parallelise the code. Without changing the orginal code, you will get error running the below cells.</mark> Click on the <b>[C++ version](../source_code/rdf.cpp)</b> or the <b>[Fortran version](../source_code/rdf.f90)</b> links, and start modifying the C or Fortran version of the RDF code. Remember to **SAVE** your code after changes, before running below cells."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compile and Run for Multicore\n",
    "\n",
    "Now, let's compile the code. We will be using NVIDIA HPC SDK for this exercise. The flags used for enabling standard parallelism for target offloading are as follows:\n",
    "\n",
    "- `-stdpar` : This flag enables standard parallelism for the target architecture\n",
    "- `-stdpar=multicore` will allow us to compile our code for a multicore\n",
    "- `-stdpar` will allow us to compile our code for a NVIDIA GPU (Default is NVIDIA)\n",
    "\n",
    "After running the cells, you can inspect part of the compiler feedback for C++ or Fortran version and see what it's telling us (your compiler feedback will be similar to the below)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### <mark>Compile the code for multicore (C++)</mark>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Compile the code for multicore (C++)\n",
    "!cd ../source_code && echo \"compiling C++ version .. \" && nvc++ -std=c++17 -stdpar=multicore -Minfo \\\n",
    "-I/opt/nvidia/hpc_sdk/Linux_x86_64/23.5/cuda/11.8/include \\\n",
    "-o rdf_c rdf.cpp -fopenmp \\\n",
    "-L/opt/nvidia/hpc_sdk/Linux_x86_64/23.5/cuda/11.8/lib64 -lnvToolsExt && ./rdf_c && cat Pair_entropy.dat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output should be the following:\n",
    "\n",
    "    \n",
    "```\n",
    "    \n",
    "s2 value is -2.43191\n",
    "s2bond value is -3.87014\n",
    "    \n",
    "```\n",
    "    \n",
    "and the compiler feedback would look similar as below:\n",
    "    \n",
    "```\n",
    "main:\n",
    "     79, stdpar: Generating Multicore code\n",
    "         79, std::fill with std::execution::par policy parallelized on CPU\n",
    "pair_gpu(double *, double *, double *, std::atomic<int> *, int, int, double, double, double, int):\n",
    "    191, stdpar: Generating Multicore code\n",
    "        191, std::for_each with std::execution::par policy parallelized on CPU\n",
    "main:\n",
    "    117, FMA (fused multiply-add) instruction(s) generated\n",
    "    146, FMA (fused multiply-add) instruction(s) generated\n",
    "    147, FMA (fused multiply-add) instruction(s) generated\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile and see output of nvptx (C++ version)\n",
    "!cd ../source_code && nsys profile -t nvtx --stats=true --force-overwrite true -o rdf_stdpar_multicore ./rdf_c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. Download and save the report file by holding down <mark>Shift</mark> and <mark>right-clicking</mark> the [C++ version](../source_code/rdf_stdpar_multicore.nsys-rep) then choosing <mark>save Link As</mark> Once done, open it via the GUI and have a look at the example expected profiler report below:\n",
    "\n",
    "**Example screenshot (C++ code)**\n",
    "\n",
    "<img src=\"../../_common/images/stdpar_multicore.png\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### <mark>Compile the code for multicore (Fortran)</mark>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Compile the code for multicore (Fortran)\n",
    "!cd ../source_code && echo \"compiling Fortran version .. \" && nvfortran -stdpar=multicore -Minfo -o rdf_f rdf.f90 -lnvhpcwrapnvtx && ./rdf_f && cat Pair_entropy.dat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Since we are targeting the NVTX v3 API, a header-only C library, and added Fortran-callable wrappers to the code, we add `-lnvhpcwrapnvtx` at the compile time to do the link to the library.\n",
    "\n",
    "The output should be the following:\n",
    "\n",
    "```\n",
    "    \n",
    "s2      :    -2.452690945278331     \n",
    "s2bond  :    -24.37502820694527 \n",
    "    \n",
    "```\n",
    "and the compiler feedback would look similar as below:\n",
    "    \n",
    "```\n",
    "rdf:\n",
    "     80, Memory zero idiom, loop replaced by call to __c_mzero8\n",
    "     92, Generating Multicore code\n",
    "         92, Loop parallelized across CPU threads\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile and see output of nvptx (Fortran version)\n",
    "!cd ../source_code && nsys profile -t nvtx --stats=true --force-overwrite true -o rdf_doconcurrent_multicore ./rdf_f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. Download and save the report file by holding down <mark>Shift</mark> and <mark>right-clicking</mark> the  [Fortran version](../source_code/rdf_doconcurrent_multicore.nsys-rep) then choosing <mark>save Link As</mark> Once done, open it via the GUI and have a look at the example expected profiler report below:\n",
    "\n",
    "**Example screenshot (Fortran code)**\n",
    "    \n",
    "<img src=\"../../_common/images/do_concurrent_multicore.jpg\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compile and run for NVIDIA GPU\n",
    "\n",
    "Without changing the code now let us try to recompile the code for NVIDIA GPU and rerun. GPU acceleration of standard parallel algorithms is enabled with the `-⁠stdpar` command-line option when using NVIDIA HPC Fortran or C++ compiler. If `-⁠stdpar `is specified, almost all algorithms that use a parallel execution policy are compiled for offloading to run in parallel on an NVIDIA GPU.\n",
    "\n",
    "**Understand and analyze** the solutions for [C++](../source_code/SOLUTION/rdf.cpp) and [Fortran](../source_code/SOLUTION/rdf.f90) versions and compare with your versions. Once done, compile your code by running below cells."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure to validate the output by running the executable and validate the output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### <mark>Compile for Tesla GPU (ISO C++)</mark>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#compile for Tesla GPU (ISO C++)\n",
    "!cd ../source_code && echo \"compiling C++ version .. \" && nvc++ -std=c++17 -DUSE_COUNTING_ITERATOR  -stdpar=gpu -Minfo -o rdf_c rdf.cpp "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The output should be the following:\n",
    "   \n",
    "```\n",
    "    \n",
    "s2 value is -2.43191\n",
    "s2bond value is -3.87014\n",
    "    \n",
    "```\n",
    "    \n",
    "and the compiler feedback would look similar as below:\n",
    "    \n",
    "```\n",
    "main:\n",
    "     79, stdpar: Generating NVIDIA GPU code\n",
    "         79, std::fill with std::execution::par policy parallelized on GPU\n",
    "pair_gpu(double *, double *, double *, std::atomic<int> *, int, int, double, double, double, int):\n",
    "    188, stdpar: Generating NVIDIA GPU code\n",
    "        188, std::for_each with std::execution::par policy parallelized on GPU\n",
    "main:\n",
    "     16, include \"nvToolsExt.h\"\n",
    "        1494, include \"nvtxImpl.h\"\n",
    "              119, FMA (fused multiply-add) instruction(s) generated\n",
    "              148, FMA (fused multiply-add) instruction(s) generated\n",
    "              149, FMA (fused multiply-add) instruction(s) generated\n",
    "pair_gpu(double *, double *, double *, std::atomic<int> *, int, int, double, double, double, int)::[lambda(unsigned int) (instance 1)]::operator ()(unsigned int) const:\n",
    "     16, include \"nvToolsExt.h\"\n",
    "        1494, include \"nvtxImpl.h\"\n",
    "              200, FMA (fused multiply-add) instruction(s) generated\n",
    "              201, FMA (fused multiply-add) instruction(s) generated\n",
    "              202, FMA (fused multiply-add) instruction(s) generated\n",
    "              204, FMA (fused multiply-add) instruction(s) generated\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile and see output of nvptx (ISO C++ version)\n",
    "!cd ../source_code && nsys profile -t nvtx,cuda --stats=true --force-overwrite true -o rdf_stdpar_gpu ./rdf_c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. Download and save the report file by holding down <mark>Shift</mark> and <mark>right-clicking</mark> the [C++ version](../source_code/rdf_stdpar_gpu.nsys-rep) then choosing <mark>save Link As</mark> Once done, open it via the GUI and have a look at the example expected profiler report below:\n",
    "\n",
    "**Example screenshot (ISO C++ code)**\n",
    "\n",
    "<img src=\"../../_common/images/stdpar_gpu.png\">\n",
    "\n",
    "If you inspect the output of the profiler closer, you can see the *Unified Memory* usage. Moreover, if you compare the NVTX marker `Pair_Calculation` (from the NVTX row) in both multicore and GPU version, you can see how much improvement you achieved. \n",
    "\n",
    "Feel free to checkout the solution for [ISO C++](../source_code/SOLUTION/rdf.cpp) version to help you understand better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### <mark>Compile for Tesla GPU (ISO Fortran)</mark>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#compile for Tesla GPU (ISO Fortran)\n",
    "!cd ../source_code &&  echo \"compiling Fortran version .. \"  && nvfortran -stdpar=gpu -Minfo -acc -o rdf_f rdf.f90 -lnvhpcwrapnvtx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Since we are targeting the NVTX v3 API, a header-only C library, and added Fortran-callable wrappers to the code, we add `-lnvhpcwrapnvtx` at the compile time to do the link to the library.\n",
    "\n",
    "The output should be the following:\n",
    "\n",
    "```\n",
    "    \n",
    "s2      :    -2.452690945278331     \n",
    "s2bond  :    -24.37502820694527 \n",
    "    \n",
    "```\n",
    "and the compiler feedback would look similar as below:\n",
    "    \n",
    "```rdf:\n",
    "     80, Memory zero idiom, loop replaced by call to __c_mzero8\n",
    "     92, Generating Tesla code\n",
    "         92, Loop parallelized across CUDA thread blocks, CUDA threads(4) blockidx%y threadidx%y\n",
    "             Loop parallelized across CUDA thread blocks, CUDA threads(32) ! blockidx%x threadidx%x\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile and see output of nvptx (ISO Fortran version)\n",
    "!cd ../source_code && nsys profile -t nvtx,cuda --stats=true --force-overwrite true -o rdf_doconcurrent_gpu ./rdf_f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. Download and save the report file by holding down <mark>Shift</mark> and <mark>right-clicking</mark> the  [Fortran version](../source_code/rdf_doconcurrent_gpu.nsys-rep) then choosing <mark>save Link As</mark> Once done, open it via the GUI and have a look at the example expected profiler report below:\n",
    "\n",
    "**Example screenshot (ISO Fortran code)**\n",
    "    \n",
    "<img src=\"../../_common/images/do_concurrent_gpu.jpg\">\n",
    "\n",
    "\n",
    "If you inspect the output of the profiler closer, you can see the *Unified Memory* usage. Moreover, if you compare the NVTX marker `Pair_Calculation` (from the NVTX row) in both multicore and GPU version, you can see how much improvement you achieved. \n",
    "\n",
    "Feel free to checkout the solutions for [ISO Fortran](../source_code/SOLUTION/rdf.f90) version to help you understand better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# stdpar Analysis\n",
    "\n",
    "**Usage Scenarios**\n",
    "- stdpar is part of the standard language, all C++ compilers are expected to support it moving forward. This is C++ standard-compliant code, so you can maintain a single codebase. It provides a good start for accelerating code on accelerators like GPU and multicores.\n",
    "- DO-CONCURRENT is part of the standard language and provides a good start for accelerating code on accelerators like GPU and multicores.\n",
    "\n",
    "**Limitations/Constraints**\n",
    "1. This isn’t a catch-all solution or necessarily an alternative to other programming models that provide more control over things like thread management. *std:par* and *DO CONCURRENT* provide the highest portability and can be seen as the first step to porting on GPU. The general abstraction limits the optimization functionalities. For example, the implementations are currently dependent on Unified memory. Moreover, one does not have control over thread management and that will limit performance improvement.\n",
    "2. C++ constructs can only be used in the code using C++17 features and may not work for legacy codes.\n",
    "\n",
    "**Which Compilers Support stdpar on GPUs and Multicore?**\n",
    "1. NVIDIA GPU: As of Jan 2021, the HPC SDK compiler from NVIDIA supports std::par and DO-CONCURRENT on NVIDIA GPU.\n",
    "2. x86 Multicore: \n",
    "    - stdpar: GCC has an implementation on a multicore CPU which is based on Intel TBB in the backend\n",
    "    - DO CONCURRENT: Other compilers like intel compiler have an implementation on a multicore CPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Post-Lab Summary\n",
    "\n",
    "If you would like to download this lab for later viewing, it is recommended you go to your browser's file menu (not the Jupyter notebook file menu) and save the complete web page.  This will ensure the images are copied down as well. You can also execute the following cell block to create a zip file of the files you have been working on, and download it with the link below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "cd ..\n",
    "rm -f _files.zip\n",
    "zip -r _files.zip *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**After** executing the above zip command, you should be able to download and save the zip file by holding down <mark>Shift</mark> and <mark>right-clicking</mark> [Here](../_files.zip) then choosing <mark>save Link As</mark>.\n",
    "\n",
    "<!--Let us now go back to parallelizing our code using other approaches.\n",
    "\n",
    "**IMPORTANT**: Please click on **HOME** to go back to the main notebook for *N ways of GPU programming for MD* code.\n",
    "\n",
    "-----\n",
    "\n",
    "# <p style=\"text-align:center;border:3px; border-style:solid; border-color:#FF0000  ; padding: 1em\"> <a href=../../../nways_MD_start.ipynb>HOME</a></p>\n",
    "\n",
    "-----\n",
    "-->\n",
    "\n",
    "# Links and Resources\n",
    "[Blog post on Developing Accelerated Code with Standard Language Parallelism](https://developer.nvidia.com/blog/developing-accelerated-code-with-standard-language-parallelism/)\n",
    "\n",
    "[Blog post on Accelerating Standard C++ with GPUs Using stdpar](https://developer.nvidia.com/blog/accelerating-standard-c-with-gpus-using-stdpar/)\n",
    "\n",
    "[Blog post on Accelerating Fortran DO CONCURRENT with GPUs and the NVIDIA HPC SDK](https://developer.nvidia.com/blog/accelerating-fortran-do-concurrent-with-gpus-and-the-nvidia-hpc-sdk/)\n",
    "\n",
    "[NVIDIA Nsight System](https://docs.nvidia.com/nsight-systems/)\n",
    "\n",
    "\n",
    "**NOTE**: To be able to see the Nsight Systems profiler output, please download the latest version of Nsight Systems from [here](https://developer.nvidia.com/nsight-systems).\n",
    "\n",
    "Don't forget to check out additional [Open Hackathons Resources](https://www.openhackathons.org/s/technical-resources) and join our [OpenACC and Hackathons Slack Channel](https://www.openacc.org/community#slack) to share your experience and get more help from the community.\n",
    "\n",
    "--- "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Licensing \n",
    "\n",
    "Copyright © 2022 OpenACC-Standard.org.  This material is released by OpenACC-Standard.org, in collaboration with NVIDIA Corporation, under the Creative Commons Attribution 4.0 International (CC BY 4.0). These materials may include references to hardware and software developed by other entities; all applicable licensing and copyrights apply."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
